Experience Report: Business Automation with Distributed Objects

A large financial institution was faced with the challenge of having to process twice the normal yearly workload without increasing their workforce. The challenge was met by reusing components of an existing object oriented application in a distributed, fully automated configuration. That solution is discussed, along with subsequent system architecture evolution, resulting improvements to development practices, and the ripple effects of changing the culture of senior management that it was designed to serve. Introduction The Ontario Teachers' Pension Plan, with over $73 billion in assets, is the single largest pension plan in Canada. It is responsible for the pension income of approximately 153,000 elementary and secondary school teachers, and 77,000 retired teachers and their families. The customer service department enjoys a reputation within the pension industry for using leading-edge information technology to provide direct and strategic support to internal core businesses, resulting in exemplary service to plan members The Benefit Entitlement System for Teachers (BEST) has been in production since December 1995. It is an internally developed Smalltalk application that makes extensive use of objectoriented practices, such as architectural layering and automated regression testing. The focus of BEST is on business process automation and system integration. For example, the system has the ability to automatically detect an incoming phone call from a client, pre-fetch the necessary data and present it to the customer service agent. At the request of the client, the agent has the ability to generate personalized estimates with several different retirement scenarios. Using our workflow system, an electronic version of an estimate letter is passed to our Quality Assurance department for manual double-checking. Once approved, it is passed to the mailroom where the document is electronically archived, printed, and mailed. In April of 1998 an early retirement incentive for teachers was announced. We were faced with the prospect of processing twice as many retirements as any previous year and handling many more phone calls and requests for personalized estimates. Failure to deal with the increased demand for retirements would have resulted in serious financial hardship for some of our clients. Our employees require twelve months of specialized training to become proficient, so adding more people wasn’t an option. What follows is a description of how objects were used to solve the problem and how the solution resulted in several unexpected changes. The system architecture changed from client-server to distributed objects; our development practices shifted from automating individual steps in a business process to automating full business processes; and the solution had the ripple effect of changing the culture of our company including its senior management. The Initial Solution When the early retirement announcement was made, the development group suggested automating the processing of these retirements as a way of meeting the increased workload. Existing objects would be reused because regression tests that tested the end-to-end processing were running cleanly. However, senior management was concerned because pensions would be paid without human intervention. This was a valid concern because in general people only retire once in their lifetime so getting their pension right the first time is very important. The risks were deemed to be manageable because low risk cases were selected, very few problems had ever been detected interactively, and a system to automate checking pensions was already on the drawing board. Once authorized, a solution was assembled that used an external workflow system for work distribution, reused domain objects to calculate and process the pensions, and leveraged a custom correspondence framework to generate a confirmation letter. All the necessary logic was built into a specialized application that ran on several computers. We made several observations. Using our workflow system to delegate work was very problematic because each program was working with cached workflow data that quickly become stale. This caused contention problems such as one case being simultaneously processed by two programs. Stressing the objects continuously raised problems with garbage collection that were corrected by programmatically activating the garbage collector. The increased database contention caused by many clients updating the same tables also exposed a bug that resulted in duplicate relational keys being assigned. Even with these problems, the project was an overall success because each of the 8,000 teachers who retired that summer was paid on time. After repeating the process the following summer, it became obvious that full automation of this process should become part of day-to-day business, not just at peak periods. What follows are summaries of how the system architecture, development practices, and the business culture have adapted to the new focus on full automation. Changes to System Architecture The system architecture of BEST changed from a fat client-server model to an architecture based on a network of lightweight components distributed across the network. The main force behind the transition was the creation of a framework for task scheduling that used a distributed infrastructure. While migrating towards a distributed architecture several interesting problems arose. To solve these problems, objects needed to be enhanced to become more resilient to the forces of distribution. The goal is to leverage our existing desktop computing resources with minimal changes to our client application. This has led to our Dispatcher/Executor framework. The Dispatcher is a broker residing on a remote host that is responsible for retrieving and dispatching work to a pool of Executor enabled clients. The Executor is an extension to the desktop client code giving it the ability to participate in a distributed architecture. It is able to receive jobs from a remote Dispatcher, and delegate them appropriately to our business logic without any user intervention. Upon completion of a job, the Executor notifies its Dispatcher that it is available for more work. Dispatcher/Executor intercommunication was achieved using a vendor-supplied CORBA solution. This mechanism is being reused to web-enable our internal applications. Figure 1: The Dispatcher/Executor framework The process of deploying systems with distributed objects raised several new issues that previously hadn’t been a problem in our client-server model. As more projects automated different tasks, it became clear that several different modes of process activation were required. The activation modes we currently support in production are: event based, scheduled, scavenging, and ad-hoc. Network hardware and software configuration problems had a fatal effect on our system. Even though our physical distribution boundaries consisted of multiple floors within one building, network problems caused inter-object communication failures, primarily due to timeouts. Application-level re-try was considered, but discarded because our strategy was to assume the worst possible case of partial failure and build facilities for easy detection and reporting of problems. It was felt that re-try could compound problems as easily as fix them. Moving from a single-threaded client environment to a multi-threaded environment forced us to protect objects that weren’t thread-safe. In particular, our database façade singleton was not thread-safe and had to be protected with a semaphore. Our objects had to become more robust as they were distributed. Objects had to become selfdiagnostic because we discovered that even one runaway server object could sabotage an entire process. For example, objects that couldn’t distinguish between a query that returned nothing because there was no database connection and a query that returned nothing because nothing matched the search criteria. To correct the problem logic was added to check the state of critical Jobs Executor Pool (Desktop PC's)